I think we'll definitely want to use individual, parallel jobs here.

I think we can ensure the build step is only performed once and reused in every job by making it a container action, something like:

# ./.github/actions/build/action.yml
name: 'Build'
description: 'Build our Docker image'
runs:
  using: 'docker'
  image: 'Dockerfile'

Then, instead of building the image with make build, we can use the jobs.<job_id>.steps.uses syntax to specify the action:

jobs:
  test:
    steps:
      - name: Check out repository
        uses: actions/checkout@v2
      - name: Run the tests
        uses: ./.github/actions/dockerfile
        with:
          entrypoint: bin/tests --postgresql-host ${ not_sure_about_this}

Our setup is a little weird right now where some makefile commands run commands in a container via docker-compose, and some run on the host. Basically whatever the script value for Travis is what we'll want to use as the entrypoint value for the workflow.

The postgres service container should only be necessary for the job that runs bin/tests. I'm not totally sure how we pass host/port/credentials to the job, it might just be that we can assume 127.0.0.1 and default credentials, like on Travis.

Hi @pavanagrawal123, wanted to check in here and see how things are going. Are you stuck on anything? Anything I can do to help? Let me know!

Hey @di,

This is actually the first time I'm seeing your other comment! I didn't get a notification for it, but this second one seemed to have triggered the GH app. I'll look into what you were saying in the next few days!

Pavan

Hey @di,

I was just testing the suggested method, and it seems to work with one caveat: we would be rebuilding the docker image on every single "job" that runs. I.e. for linting, testing, etc, we would be building the Docker image every time. This doesn't actually come with too many downsides, it seems like the only downsides are that GH's compute resources are going to be wasted. Should we make an effort to figure out how to cache the build between jobs so we don't waste these compute resources? I think that pushing an image as a build artifact may be one way we can achieve this, but it may end up being kind of messy. Let me know what you think!

(The docker build takes about 6 minutes of compute time, so we would be wasting about 42 compute minutes for each push)

Pavan

Relevant here: https://dev.to/dtinth/caching-docker-builds-in-github-actions-which-approach-is-the-fastest-a-research-18ei

@pradyunsg , thanks that was really insightful!

It looks like the GitHub package repository approach or buildkit approach look ideal. We can use that in the first job and then have the subsequent jobs need the first job.

If we can get a few confirmations on if this seems like the right way forward, I'll work on this method.

Sounds good to me!

Thanks @pradyunsg! I'd be most interested in a solution that would allow us to cache intermediate containers between CI runs as well, as most code changes aren't going to require rebuilding the entire image, just the last few layers.

@di, I agree that's ideal implementation. I'm not super familiar with the ins and outs of docker's build process and how it handles caching. It appears that the article mentions that utilizing the Github Package Repository will give us layer-by-layer caching. Do you think this is the right approach @di?

Thanks!

Yep, I think it's most likely the best approach.

Yea, basically the caching with GitHub Packages Registry should do what we want here, as far as I can tell.

We'll also want to make sure to name stuff correctly, so that folks don't try to use the CI containers for some reason. (oh, I can just use that)

My apologies for being so delayed on this issue. I am currently running into some issues, again with caching on GH image repository, but I hope to have those issues resolved soon!